When Is There a Free Matrix Lunch?

The “no-free lunch theorems” essentially say that for any two algorithms A and B, there are “as many” targets (or priors over targets) for which A has lower expected loss than B as vice-versa. This can be made precise for certain loss functions [WM97]. This note concerns itself with cases where seemingly harder matrix versions of the algorithms have the same on-line loss bounds as the corresponding vector versions. So it seems that you get a free “matrix lunch” (Our title is however not meant to imply that we have a technical refutation of the no-free lunch theorems). The simplest case of this phenomenon occurs in the so-called expert setting. We have n experts. In each trial the algorithm proposes a probability vector w over the n experts, receives a loss vector ` ∈ [0, 1] for the experts and incurs an expected loss w ·`. The Weighted Majority or Hedge algorithm uses exponential weights w i ∼ w i e−η Pt−1 t=1 ` t i and has the following expected loss bound [FS97]: